Equivalence of Entropy Regularization and Relative-Entropy Proximal Method

We consider two entropy-based interior point methods that solve LP relaxations of MAP estimation in graphical models: (1) an entropy-regularization method and (2) a relative-entropy proximal method. Using the fact that relative-entropy is the Bregman distance induced by entropy, we show that the two approaches are actually equivalent. The purpose of this note is to show one connection between the two approaches described in [1, 2]. Another connection between these two works is that both use distributed iterative-scaling/Bregman-projections algorithms [3, 4] to solve the “inner-loop” optimizations required by the methods summarized below. This second connection, however, is not explored in this present note. Introduction For the sake of this note, we consider the exponential family of probability distributions on n binary variables x = (x1, . . . , xn) ∈ {0, 1} : P (x) = exp{θφ(x) − Φ(θ)} = 1 Z(θ) exp{ ∑